<html>
<head><meta charset="utf-8"><title>crater performance · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html">crater performance</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="234573959"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234573959" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234573959">(Apr 14 2021 at 21:00)</a>:</h4>
<p>FWIW, I've been doing some investigation on whether the crater boxes are currently 'well tuned' for our workload, though I admit it's been sort of inbetween other tasks and not particularly well researched. (Maybe folks have some pointers on good documentation here; I wasn't really able to find anything that looked recently updated and easy-ish to use).</p>
<p>One thing I did notice is that the GCP machines are context switching a <em>lot</em> more - 3x - compared to azure machines. Azure has context-switches at 0.249 K/sec, according to perf, while the gcp ones are at 0.684 K/sec.</p>
<p>I don't really know if there's an easy way to compare two systems - with different core counts, too - in a way that nicely tracks the cause of context switches. Obviously, the workloads in terms of what crates are getting scheduled and such are different too; this might be caused by that. I only measured across 5 minutes - a longer measurement might be in order to smooth out some of this, but I suspect it won't change anything.</p>
<p>cc <span class="user-mention" data-user-id="239881">@Josh Triplett</span> - I'm wondering if you happen to know knobs we should be tuning to optimize for crater-style workloads, or whether the current strategy of overloading the system (e.g., we have 60 rustc's running right now, with only 14 cores) is a good idea. Maybe there's some documentation I wasn't able to find.</p>
<p>I've tried adjusting sched_min_granularity_ns up to 1s, but this seemed to not change anything that I could see; I don't know whether it just had no effect or if something else was causing the problem.</p>



<a name="234574940"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234574940" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234574940">(Apr 14 2021 at 21:03)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> Are you using the default kernels, or custom kernels? Ideally, I'd suggest using near-identical kernel images on both systems for a comparison like that.</p>



<a name="234574976"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234574976" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234574976">(Apr 14 2021 at 21:03)</a>:</h4>
<p>Can you grab the kernel configurations from both, and I'll diff them and look for notable issues?</p>



<a name="234575132"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234575132" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234575132">(Apr 14 2021 at 21:04)</a>:</h4>
<p>They are different kernel versions (one is 5.0.0, though customized by azure), the other is 5.4</p>



<a name="234575190"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234575190" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234575190">(Apr 14 2021 at 21:04)</a>:</h4>
<p>Regarding overloads, I personally would favor having only as many rustc instances as logical CPUs. (Threads, not cores.) I can imagine scenarios where having a few more would be useful, but not 4x as many.</p>



<a name="234575223"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234575223" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234575223">(Apr 14 2021 at 21:04)</a>:</h4>
<p>in terms of grabbing configuration, I guess you mean sysctl -A or something?</p>



<a name="234575260"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234575260" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234575260">(Apr 14 2021 at 21:04)</a>:</h4>
<p>No, I mean the kconfig compile-time kernel configuration.</p>



<a name="234575318"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234575318" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234575318">(Apr 14 2021 at 21:05)</a>:</h4>
<p>Commonly found in /boot/config-(thekernelversion), or potentially /proc/config.gz</p>



<a name="234575381"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234575381" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234575381">(Apr 14 2021 at 21:05)</a>:</h4>
<p>All the <code>CONFIG_XYZ</code> options.</p>



<a name="234575631"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234575631" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234575631">(Apr 14 2021 at 21:06)</a>:</h4>
<p><a href="/user_uploads/4715/dnU5KH9hmqG261k3lXVp7frJ/gcp">gcp</a> <a href="/user_uploads/4715/RhR9lm47y_lHlJKZS9z0DQN6/azure">azure</a></p>



<a name="234575662"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234575662" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234575662">(Apr 14 2021 at 21:06)</a>:</h4>
<p><span class="user-mention silent" data-user-id="239881">Josh Triplett</span> <a href="#narrow/stream/242791-t-infra/topic/crater.20performance/near/234575190">said</a>:</p>
<blockquote>
<p>Regarding overloads, I personally would favor having only as many rustc instances as logical CPUs. (Threads, not cores.) I can imagine scenarios where having a few more would be useful, but not 4x as many.</p>
</blockquote>
<p>so, the reason why crater currently overloads is to "fill" the downtime between rustc executions, as crater also does a lot of non-compilation</p>



<a name="234575791"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234575791" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234575791">(Apr 14 2021 at 21:06)</a>:</h4>
<p>4x is probably too much indeed, but it's not like crater is compiling all the time</p>



<a name="234575799"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234575799" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234575799">(Apr 14 2021 at 21:06)</a>:</h4>
<p><span class="user-mention" data-user-id="121055">@Pietro Albini</span> Ah, interesting.</p>



<a name="234575823"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234575823" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234575823">(Apr 14 2021 at 21:07)</a>:</h4>
<p>I wonder if we could measure the % of time spent in rustc/linkers vs. other stuff</p>



<a name="234575979"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234575979" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234575979">(Apr 14 2021 at 21:07)</a>:</h4>
<p>It's not like 4x is going to make it that much <em>worse</em>; the kernel is relatively good at context switching. We're talking a few percent potential performance loss, not an order of magnitude.</p>



<a name="234576013"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234576013" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234576013">(Apr 14 2021 at 21:07)</a>:</h4>
<p>As long as you aren't overloading the amount of memory on the system.</p>



<a name="234576041"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234576041" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234576041">(Apr 14 2021 at 21:08)</a>:</h4>
<p>Yeah, no, plenty of free memory.</p>



<a name="234576265"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234576265" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234576265">(Apr 14 2021 at 21:08)</a>:</h4>
<p>On average the azure machine has 20 jobs/minute over 48 hour period and gcp has 15 jobs/minute; gcp has 14 cores vs 16 on the azure ones (vCPUs, so, logical cores)</p>



<a name="234576299"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234576299" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234576299">(Apr 14 2021 at 21:08)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/crater.20performance/near/234575823">said</a>:</p>
<blockquote>
<p>I wonder if we could measure the % of time spent in rustc/linkers vs. other stuff</p>
</blockquote>
<p>I guess? it'd be quite hard to measure that though, since for every rustc execution crater does the equivalent of <code>docker run --rm -it crates-build-env cargo build</code>, so we'd not be able to measure the time needed to spin up the docker container</p>



<a name="234576422"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234576422" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234576422">(Apr 14 2021 at 21:09)</a>:</h4>
<p>So there's a 17% performance delta "per core" between them, looks like, roughly</p>



<a name="234576487"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234576487" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234576487">(Apr 14 2021 at 21:09)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/crater.20performance/near/234576265">said</a>:</p>
<blockquote>
<p>On average the azure machine has 20 jobs/minute over 48 hour period and gcp has 15 jobs/minute; gcp has 14 cores vs 16 on the azure ones (vCPUs, so, logical cores)</p>
</blockquote>
<p>crater also does a bunch of IO, and in my experience locally disk performance impacted crater a bit</p>



<a name="234576643"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234576643" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234576643">(Apr 14 2021 at 21:10)</a>:</h4>
<p>(running crater with the workspace on hdd vs ssd makes a huge difference, and even ssd vs nvme has an impact)</p>



<a name="234576662"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234576662" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234576662">(Apr 14 2021 at 21:10)</a>:</h4>
<p>It's mostly writing based on our metrics</p>



<a name="234576847"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234576847" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234576847">(Apr 14 2021 at 21:11)</a>:</h4>
<p>"plenty of free memory" -- enough to give them ram-disks?</p>



<a name="234576911"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234576911" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234576911">(Apr 14 2021 at 21:11)</a>:</h4>
<p>not <em>that</em> much :)</p>



<a name="234576980"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234576980" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234576980">(Apr 14 2021 at 21:11)</a>:</h4>
<p>another difference between azure and gcp is the disk space, with gcp having 300gb disks and azure having 2tb disks</p>



<a name="234577129"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234577129" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234577129">(Apr 14 2021 at 21:12)</a>:</h4>
<p>and that means gcp has to erase the cached prebuilt dependencies more often</p>



<a name="234577247"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234577247" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234577247">(Apr 14 2021 at 21:12)</a>:</h4>
<p>47 MB/s write / 15.5 MB/s read on average on azure</p>



<a name="234577453"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234577453" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234577453">(Apr 14 2021 at 21:13)</a>:</h4>
<p>Interestingly, GCP is 38 MB/s write and only 1.4 MB/s read</p>



<a name="234577469"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234577469" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234577469">(Apr 14 2021 at 21:13)</a>:</h4>
<p>are they built with debuginfo? could reduce that</p>



<a name="234577531"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234577531" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234577531">(Apr 14 2021 at 21:13)</a>:</h4>
<p>though full beta-crater should probably not cut corners</p>



<a name="234577625"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234577625" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234577625">(Apr 14 2021 at 21:14)</a>:</h4>
<p>most runs are check-only, so most of the times we shouldn't go to codegen</p>



<a name="234577646"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234577646" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234577646">(Apr 14 2021 at 21:14)</a>:</h4>
<p>Should all be the same</p>



<a name="234577661"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234577661" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234577661">(Apr 14 2021 at 21:14)</a>:</h4>
<p>Right now a check run</p>



<a name="234577749"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234577749" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234577749">(Apr 14 2021 at 21:14)</a>:</h4>
<p>ah, would only be build-deps then</p>



<a name="234578045"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234578045" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234578045">(Apr 14 2021 at 21:16)</a>:</h4>
<p>also, for context for who's not familiar with crater internals, to cache dependencies between crates each crater thread has a single <code>target</code> directory shared between all builds run by that thread</p>



<a name="234578088"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234578088" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234578088">(Apr 14 2021 at 21:16)</a>:</h4>
<p>when the disk is close to full all those target directories are <code>rm -rf</code>'d, pruning the caches</p>



<a name="234578143"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234578143" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234578143">(Apr 14 2021 at 21:17)</a>:</h4>
<p>so it wouldn't surprise me that having 300gb of scratch disk space compared to 2tb would have an impact on the overall number of jobs completed by crater</p>



<a name="234578194"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234578194" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234578194">(Apr 14 2021 at 21:17)</a>:</h4>
<p>couldn't sccache handle that more gracefully, like LRU?</p>



<a name="234578322"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234578322" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234578322">(Apr 14 2021 at 21:18)</a>:</h4>
<p>I don't think we've tried doing sccache on crater, we could even do S3 with that in theory, though working out how to get credentials down there would be annoying</p>



<a name="234578429"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234578429" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234578429">(Apr 14 2021 at 21:19)</a>:</h4>
<p>hmm, doesn't sccache not work that well when passing different build flags to it?</p>



<a name="234578535"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234578535" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234578535">(Apr 14 2021 at 21:20)</a>:</h4>
<p>It doesn't support base directory, but I think otherwise works well?</p>



<a name="234578602"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234578602" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234578602">(Apr 14 2021 at 21:20)</a>:</h4>
<p>And for our use case it's all in docker at a common directory</p>



<a name="234578909"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234578909" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234578909">(Apr 14 2021 at 21:23)</a>:</h4>
<p>There are a few notable kernel configuration differences, but nothing worth building and maintaining a custom kernel for.</p>



<a name="234579534"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234579534" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234579534">(Apr 14 2021 at 21:27)</a>:</h4>
<p>Yeah. I want to at least update them to be the same version, but I doubt that would be a major difference - especially because the older one is actually faster right now. It might well be just the disk size difference as Pietro notes, though.</p>



<a name="234579766"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234579766" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234579766">(Apr 14 2021 at 21:29)</a>:</h4>
<p>a practical thing we could do to make slight improvements would be to add a <code>docker image prune</code> cronjob on the agents, to delete the old docker images</p>



<a name="234579790"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234579790" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234579790">(Apr 14 2021 at 21:29)</a>:</h4>
<p>that command freed up 40gb of storage on gcp-1</p>



<a name="234579809"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234579809" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234579809">(Apr 14 2021 at 21:29)</a>:</h4>
<p>(crates-build-env is around 7gb in size)</p>



<a name="234579959"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234579959" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234579959">(Apr 14 2021 at 21:30)</a>:</h4>
<p>Hm, yes, that seems promising - I guess that storage should be used for caches now?</p>



<a name="234579974"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234579974" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234579974">(Apr 14 2021 at 21:30)</a>:</h4>
<p>yep!</p>



<a name="234580266"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234580266" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234580266">(Apr 14 2021 at 21:33)</a>:</h4>
<p>right now crater doesn't remove caches until the total filesystem utilization reaches 85%</p>



<a name="234580300"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234580300" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234580300">(Apr 14 2021 at 21:33)</a>:</h4>
<p>(it might actually delete stuff a bit later than when it reaches 85%, depending on how long the current crates finishes building)</p>



<a name="234580415"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234580415" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234580415">(Apr 14 2021 at 21:34)</a>:</h4>
<p>whelp on gcp-2 I free'd 90gb of stale images</p>



<a name="234580529"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234580529" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234580529">(Apr 14 2021 at 21:35)</a>:</h4>
<p>Good to know. I can file a PR tomorrow, and we should probably add some monitoring for disk space used by the target directories (measured when deleting them).</p>



<a name="234580781"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234580781" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234580781">(Apr 14 2021 at 21:37)</a>:</h4>
<p>It might be worth stopping things, clearing out the target directories like crater does, and then checking how much space we're using for other stuff</p>



<a name="234580811"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234580811" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234580811">(Apr 14 2021 at 21:37)</a>:</h4>
<p>(maybe some of that base image load can be deleted)</p>



<a name="234581037"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234581037" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234581037">(Apr 14 2021 at 21:39)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/crater.20performance/near/234580781">said</a>:</p>
<blockquote>
<p>It might be worth stopping things, clearing out the target directories like crater does, and then checking how much space we're using for other stuff</p>
</blockquote>
<p>hmm, we already kinda know that</p>



<a name="234581132"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234581132" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234581132">(Apr 14 2021 at 21:40)</a>:</h4>
<p><a href="/user_uploads/4715/6i0ZnvG8AOO6-xDTUcx1sqBJ/2021-04-14-23-39-59.png">2021-04-14-23-39-59.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/6i0ZnvG8AOO6-xDTUcx1sqBJ/2021-04-14-23-39-59.png" title="2021-04-14-23-39-59.png"><img src="/user_uploads/4715/6i0ZnvG8AOO6-xDTUcx1sqBJ/2021-04-14-23-39-59.png"></a></div>



<a name="234581191"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234581191" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234581191">(Apr 14 2021 at 21:40)</a>:</h4>
<p>the dips here should be as soon as crater finished purging the caches</p>



<a name="234581211"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234581211" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234581211">(Apr 14 2021 at 21:40)</a>:</h4>
<p>(that was gcp-1)</p>



<a name="234581255"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234581255" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234581255">(Apr 14 2021 at 21:41)</a>:</h4>
<p><a href="/user_uploads/4715/3N-m7EyoFKRhYGFaUj1ZgMcz/2021-04-14-23-41-08.png">2021-04-14-23-41-08.png</a> <br>
while this is azure-1</p>
<div class="message_inline_image"><a href="/user_uploads/4715/3N-m7EyoFKRhYGFaUj1ZgMcz/2021-04-14-23-41-08.png" title="2021-04-14-23-41-08.png"><img src="/user_uploads/4715/3N-m7EyoFKRhYGFaUj1ZgMcz/2021-04-14-23-41-08.png"></a></div>



<a name="234581677"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234581677" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234581677">(Apr 14 2021 at 21:44)</a>:</h4>
<p>That seems most likely to be the culprit of performance differences.</p>



<a name="234581704"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234581704" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234581704">(Apr 14 2021 at 21:45)</a>:</h4>
<p>Would it be worth testing a larger disk on GCP to see if it improves performance substantially?</p>



<a name="234581898"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234581898" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234581898">(Apr 14 2021 at 21:46)</a>:</h4>
<p>iirc I chose 300gb because that was exactly within the budget we had on gcp</p>



<a name="234581922"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234581922" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234581922">(Apr 14 2021 at 21:46)</a>:</h4>
<p>otherwise we'd have to remove some cores, or well, pay a bit more :)</p>



<a name="234582218"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582218" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582218">(Apr 14 2021 at 21:49)</a>:</h4>
<p>Hm, why do we have 500gb left on the azure machines then? That seems really excessive after purging</p>



<a name="234582248"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582248" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582248">(Apr 14 2021 at 21:49)</a>:</h4>
<p>running <code>docker purge</code> on azure-1</p>



<a name="234582252"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582252" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582252">(Apr 14 2021 at 21:49)</a>:</h4>
<p>150gb on gcp also seems way too large</p>



<a name="234582344"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582344" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582344">(Apr 14 2021 at 21:50)</a>:</h4>
<p>How big is the latest docker image? I recall us having a big one</p>



<a name="234582370"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582370" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582370">(Apr 14 2021 at 21:50)</a>:</h4>
<p>7gb</p>



<a name="234582389"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582389" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582389">(Apr 14 2021 at 21:50)</a>:</h4>
<p>Ah. Ok. So not that.</p>



<a name="234582398"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582398" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582398">(Apr 14 2021 at 21:50)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/crater.20performance/near/234581922">said</a>:</p>
<blockquote>
<p>otherwise we'd have to remove some cores, or well, pay a bit more :)</p>
</blockquote>
<p>Definitely not worth trading cores for storage, I think.</p>



<a name="234582419"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582419" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582419">(Apr 14 2021 at 21:50)</a>:</h4>
<p><span class="user-mention silent" data-user-id="239881">Josh Triplett</span> <a href="#narrow/stream/242791-t-infra/topic/crater.20performance/near/234582398">said</a>:</p>
<blockquote>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/crater.20performance/near/234581922">said</a>:</p>
<blockquote>
<p>otherwise we'd have to remove some cores, or well, pay a bit more :)</p>
</blockquote>
<p>Definitely not worth trading cores for storage, I think.</p>
</blockquote>
<p>that was the call I made too</p>



<a name="234582423"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582423" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582423">(Apr 14 2021 at 21:51)</a>:</h4>
<p>I'm curious, when you say "budget we had on GCP", is that donated by Google?</p>



<a name="234582455"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582455" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582455">(Apr 14 2021 at 21:51)</a>:</h4>
<p>Yeah, we can likely increase that - it's mostly a matter of getting the most we can out of the machines we have, IMO</p>



<a name="234582467"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582467" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582467">(Apr 14 2021 at 21:51)</a>:</h4>
<p>yes, they gave us a round amount of credits and I squeezed as much as I could there</p>



<a name="234582559"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582559" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582559">(Apr 14 2021 at 21:52)</a>:</h4>
<p>One thought...</p>



<a name="234582594"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582594" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582594">(Apr 14 2021 at 21:52)</a>:</h4>
<p>If there are two GCP systems, would it be feasible to switch to one with twice the cores and twice the disk?</p>



<a name="234582608"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582608" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582608">(Apr 14 2021 at 21:52)</a>:</h4>
<p>That should perform comparably, but share more crate cache.</p>



<a name="234582629"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582629" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582629">(Apr 14 2021 at 21:53)</a>:</h4>
<p>Crate cache isn't shared between workers</p>



<a name="234582664"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582664" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582664">(Apr 14 2021 at 21:53)</a>:</h4>
<p>I don't think it would share more crates: we keep a target directory per thread, otherwise cargo would lock the build with "waiting on lock on target directory"</p>



<a name="234582674"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582674" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582674">(Apr 14 2021 at 21:53)</a>:</h4>
<p>(so e.g. the azure machines have 16 distinct caches, gcp 14 distinct caches)</p>



<a name="234582680"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582680" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582680">(Apr 14 2021 at 21:53)</a>:</h4>
<p>Oh, I didn't realize that we ran a worker per thread.</p>



<a name="234582705"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582705" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582705">(Apr 14 2021 at 21:53)</a>:</h4>
<p>iirc we <em>also</em> keep different copies of another cargo directory to avoid another lock error</p>



<a name="234582716"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582716" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582716">(Apr 14 2021 at 21:53)</a>:</h4>
<p>can't recall which off the top of my head</p>



<a name="234582787"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582787" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582787">(Apr 14 2021 at 21:54)</a>:</h4>
<p>In theory, we could run a background deduplication mechanism that hardlinked identical files between the caches, but anything more than that would be complicated, yeah.</p>



<a name="234582796"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582796" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582796">(Apr 14 2021 at 21:54)</a>:</h4>
<p>What might make sense is to switch to e.g. zfs or btrfs to have automatic content-based-deduplication of the cache, which might save space. But I'm not sure our builds are sufficiently identical for that to make sense.</p>



<a name="234582832"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582832" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582832">(Apr 14 2021 at 21:55)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/crater.20performance/near/234582389">said</a>:</p>
<blockquote>
<p>Ah. Ok. So not that.</p>
</blockquote>
<p><code>docker image prune</code> on <code>azure-1</code> did not exit yet, and it already removed 100gb</p>



<a name="234582849"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234582849" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234582849">(Apr 14 2021 at 21:55)</a>:</h4>
<p>(Or maybe I'm remembering wrong, and those FSs don't do that)</p>



<a name="234584018"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234584018" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234584018">(Apr 14 2021 at 22:04)</a>:</h4>
<p>We also have a bunch of (old) toolchains in the crater-agent-workspace, it looks like</p>



<a name="234584348"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234584348" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234584348">(Apr 14 2021 at 22:07)</a>:</h4>
<p>57 GB of them on gcp-1</p>



<a name="234584555"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234584555" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234584555">(Apr 14 2021 at 22:08)</a>:</h4>
<p>opened an issue for pruning docker stuff: <a href="https://github.com/rust-lang/crater/issues/571">https://github.com/rust-lang/crater/issues/571</a></p>



<a name="234607947"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234607947" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234607947">(Apr 15 2021 at 02:44)</a>:</h4>
<p><span class="user-mention" data-user-id="121055">@Pietro Albini</span> didn't we have a plan at some point to shrink the image by separating it into layers? So all but the most recently updated layer could be cached?</p>



<a name="234645250"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234645250" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234645250">(Apr 15 2021 at 09:38)</a>:</h4>
<p>I think we did, but I'm not convinced it's a good idea nowadays</p>



<a name="234645568"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234645568" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234645568">(Apr 15 2021 at 09:41)</a>:</h4>
<p>for production (<a href="http://docs.rs">docs.rs</a> and crater) we want that image to be up to date, and download times don't matter much</p>



<a name="234645649"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234645649" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234645649">(Apr 15 2021 at 09:41)</a>:</h4>
<p>and now rustwide supports arbitrary images, so for local development we could create a <code>crates-build-env-micro</code> with just <code>build-essential</code> and <code>libssl-dev</code></p>



<a name="234674056"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234674056" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234674056">(Apr 15 2021 at 13:26)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/crater.20performance/near/234645568">said</a>:</p>
<blockquote>
<p>for production (<a href="http://docs.rs">docs.rs</a> and crater) we want that image to be up to date, and download times don't matter much</p>
</blockquote>
<p>well I'm not worried as much about download time, but it would save a lot on disk space instead of using 7 GB on each update</p>



<a name="234674132"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234674132" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234674132">(Apr 15 2021 at 13:26)</a>:</h4>
<p>I don't see what the image being up to date has to do with it? docker does the caching itself, it would still download any newer version</p>



<a name="234700560"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234700560" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234700560">(Apr 15 2021 at 15:40)</a>:</h4>
<p>oh, I meant having the latest version of packages</p>



<a name="234835333"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234835333" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234835333">(Apr 16 2021 at 11:15)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> uh the experiment you tried needs to be tuned a bit better</p>



<a name="234835925"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234835925" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234835925">(Apr 16 2021 at 11:20)</a>:</h4>
<p><a href="/user_uploads/4715/XcmqaK3BmLYdjPN07uveWMvd/2021-04-16-13-20-43.png">2021-04-16-13-20-43.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/XcmqaK3BmLYdjPN07uveWMvd/2021-04-16-13-20-43.png" title="2021-04-16-13-20-43.png"><img src="/user_uploads/4715/XcmqaK3BmLYdjPN07uveWMvd/2021-04-16-13-20-43.png"></a></div>



<a name="234835994"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234835994" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234835994">(Apr 16 2021 at 11:21)</a>:</h4>
<p><a href="/user_uploads/4715/AXpvywiLD7KfFJJDOuOUMcDO/2021-04-16-13-21-47.png">2021-04-16-13-21-47.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/AXpvywiLD7KfFJJDOuOUMcDO/2021-04-16-13-21-47.png" title="2021-04-16-13-21-47.png"><img src="/user_uploads/4715/AXpvywiLD7KfFJJDOuOUMcDO/2021-04-16-13-21-47.png"></a></div>



<a name="234836129"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234836129" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234836129">(Apr 16 2021 at 11:22)</a>:</h4>
<p>zooming in on a single agent it seems like the threads start, load skyrockets, then everything stops until the load goes back to a reasonable amount, then <code>goto 0</code></p>



<a name="234836378"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234836378" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234836378">(Apr 16 2021 at 11:25)</a>:</h4>
<p>Oh, I was planning to be around to babysit it</p>



<a name="234836395"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234836395" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234836395">(Apr 16 2021 at 11:25)</a>:</h4>
<p>But yeah that's not unexpected</p>



<a name="234836846"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234836846" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234836846">(Apr 16 2021 at 11:28)</a>:</h4>
<p>do you want to check things in prod or should I revert?</p>



<a name="234837309"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234837309" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234837309">(Apr 16 2021 at 11:32)</a>:</h4>
<p>Let's revert</p>



<a name="234837332"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234837332" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234837332">(Apr 16 2021 at 11:33)</a>:</h4>
<p>I don't think there's any data to gain from prod</p>



<a name="234837421"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234837421" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234837421">(Apr 16 2021 at 11:33)</a>:</h4>
<p>How difficult would it be to setup a staging branch for just one of the machines, so I could experiment a bit more easily?</p>



<a name="234837551"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234837551" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234837551">(Apr 16 2021 at 11:34)</a>:</h4>
<p>hmm</p>



<a name="234837610"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234837610" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234837610">(Apr 16 2021 at 11:34)</a>:</h4>
<p>we'd need to either create a second ECR repo and point one of the machines to it</p>



<a name="234837636"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234837636" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234837636">(Apr 16 2021 at 11:35)</a>:</h4>
<p>or push the staging images to the same ECR repo but with a separate tag</p>



<a name="234837686"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234837686" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234837686">(Apr 16 2021 at 11:35)</a>:</h4>
<p>(rolled back the image with <code>simpleinfra/aws-rollback.py</code>, will open a revert PR now)</p>



<a name="234837837"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234837837" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234837837">(Apr 16 2021 at 11:36)</a>:</h4>
<p><a href="https://github.com/rust-lang/crater/pull/574">https://github.com/rust-lang/crater/pull/574</a></p>



<a name="234838545"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234838545" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234838545">(Apr 16 2021 at 11:42)</a>:</h4>
<p>I think a separate tag feels good</p>



<a name="234838602"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234838602" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234838602">(Apr 16 2021 at 11:43)</a>:</h4>
<p>separate tag is definitely the least impactful change</p>



<a name="234838633"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234838633" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234838633">(Apr 16 2021 at 11:43)</a>:</h4>
<p><a href="/user_uploads/4715/-UyRUIPv2bcJh8ZXK0yRi3Zs/2021-04-16-13-43-12.png">2021-04-16-13-43-12.png</a> <br>
but I'm a bit worried then about losing the ability to easily rollback prod</p>
<div class="message_inline_image"><a href="/user_uploads/4715/-UyRUIPv2bcJh8ZXK0yRi3Zs/2021-04-16-13-43-12.png" title="2021-04-16-13-43-12.png"><img src="/user_uploads/4715/-UyRUIPv2bcJh8ZXK0yRi3Zs/2021-04-16-13-43-12.png"></a></div>



<a name="234838693"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234838693" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234838693">(Apr 16 2021 at 11:44)</a>:</h4>
<p>(ECR is configured to only keep 4 non-tagged images)</p>



<a name="234838773"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234838773" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234838773">(Apr 16 2021 at 11:44)</a>:</h4>
<p>I guess... pros and cons for each approach</p>



<a name="234839542"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234839542" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234839542">(Apr 16 2021 at 11:51)</a>:</h4>
<p>I don't think we'd be deploying to prod while this is being donr</p>



<a name="234839642"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234839642" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234839642">(Apr 16 2021 at 11:52)</a>:</h4>
<p>I expect to finish over the weekend or so, at least with some initial work, and so I'm not too worried about the 4 item history</p>



<a name="234839669"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234839669" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234839669">(Apr 16 2021 at 11:52)</a>:</h4>
<p>We could also just manually set that to 200 for now, but doesn't seem necessary</p>



<a name="234839683"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234839683" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234839683">(Apr 16 2021 at 11:52)</a>:</h4>
<p>I can set this up if you're not opposed</p>



<a name="234839783"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234839783" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234839783">(Apr 16 2021 at 11:54)</a>:</h4>
<p><span class="user-mention" data-user-id="121055">@Pietro Albini</span> ^</p>



<a name="234841312"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234841312" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234841312">(Apr 16 2021 at 12:07)</a>:</h4>
<p>yeah that works</p>



<a name="234841330"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234841330" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234841330">(Apr 16 2021 at 12:07)</a>:</h4>
<p>at the end of the day it's not a huge deal if we can't rollback crater instantly</p>



<a name="234841404"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234841404" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234841404">(Apr 16 2021 at 12:08)</a>:</h4>
<p>yeah</p>



<a name="234841452"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234841452" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234841452">(Apr 16 2021 at 12:08)</a>:</h4>
<p>OK, I'll set that up then when I get a chance and try to drive this forward, still unclear if it'll actually help</p>



<a name="234841489"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234841489" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234841489">(Apr 16 2021 at 12:08)</a>:</h4>
<p><span aria-label="thumbs up" class="emoji emoji-1f44d" role="img" title="thumbs up">:thumbs_up:</span>.</p>



<a name="234979816"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234979816" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234979816">(Apr 17 2021 at 10:51)</a>:</h4>
<p>You may want to sample /proc/pressure/cpu instead of load average. The load average also includes tasks waiting for IO which would incur a context switch anyway. And it has higher resolution</p>



<a name="234980171"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234980171" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234980171">(Apr 17 2021 at 10:58)</a>:</h4>
<blockquote>
<p>What might make sense is to switch to e.g. zfs or btrfs to have automatic content-based-deduplication of the cache, which might save space. But I'm not sure our builds are sufficiently identical for that to make sense.</p>
</blockquote>
<p>btrfs doesn't have online deduplication. and offline deduplication via reflinks doesn't share the page cache afaik.<br>
zfs has online deduplication but it eats tons of ram<br>
hardlinks could work</p>



<a name="234980488"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234980488" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234980488">(Apr 17 2021 at 11:03)</a>:</h4>
<p>If writes are an issue there's also a bunch of filesystem durability features you can modify (e.g. noauto_da_alloc and data=writeback on ext4), <br>
And if docker uses overlayfs as storage driver then the <a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c86243b090bc25f81abe33ddfa9fe833e02c4d64">new volatile mount option in 5.10</a> might help.<br>
Not sure though if can actually be enabled via remount or has to be there from the first mount.</p>



<a name="234981816"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/234981816" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#234981816">(Apr 17 2021 at 11:27)</a>:</h4>
<p>no, doesn't look like it can be added with a remount. so another layer of overlayfs on top would be needed to use it</p>



<a name="235034844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235034844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235034844">(Apr 18 2021 at 01:59)</a>:</h4>
<p>Thanks - that's all very helpful</p>



<a name="235034925"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235034925" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235034925">(Apr 18 2021 at 02:00)</a>:</h4>
<p>I've deployed to gcp-1 usage of the jobserver that's already supported by cargo, which has reduced the load on the system; it's not clear yet (too short sample size) if this is a net improvement in throughput though. No excellent results either, though.</p>



<a name="235035606"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235035606" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235035606">(Apr 18 2021 at 02:14)</a>:</h4>
<p>we'll see how it does over the next some hours - if it seems to break, then the config for the systemd job merely needs to be edited to :latest and that should fix it.</p>
<p>My guess is it'll run into the typical jobserver bug of tokens not getting returned if something exits abnormally, but we'll see if that poses a problem in practice. It's also quite possible this is ~useless and we already work well as-is, without these changes</p>



<a name="235062696"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235062696" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235062696">(Apr 18 2021 at 10:40)</a>:</h4>
<p>cargo's jobserver doesn't benefit from recent kernel optimizations though since it uses polling instead of blocking reads. so more context switches than necessary. I already filed an issue for that... maybe I should just write a PR.</p>



<a name="235459584"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235459584" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235459584">(Apr 21 2021 at 05:58)</a>:</h4>
<p><span class="user-mention" data-user-id="330154">@The 8472</span> If there are optimizations you could offer there, that'd be highly welcome.</p>



<a name="235475795"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235475795" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235475795">(Apr 21 2021 at 08:53)</a>:</h4>
<p>They're already in jobserver-rs 0.1.22. <a href="https://github.com/alexcrichton/jobserver-rs/commits/master">https://github.com/alexcrichton/jobserver-rs/commits/master</a><br>
That's a reader side optimization. I assume cargo just passes the file descriptors it through to the rustc instances? If so the dependency in rustc needs to be bumped.</p>



<a name="235572625"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235572625" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235572625">(Apr 21 2021 at 20:03)</a>:</h4>
<p><a href="https://github.com/rust-lang/rust/issues/84407">#84407</a></p>



<a name="235573959"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235573959" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235573959">(Apr 21 2021 at 20:12)</a>:</h4>
<p>Yes, and regardless, the Cargo lockfile is either ignored or doesn't exist (I forget) for the cargo's we build.</p>



<a name="235575488"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235575488" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235575488">(Apr 21 2021 at 20:25)</a>:</h4>
<blockquote>
<p>My guess is it'll run into the typical jobserver bug of tokens not getting returned if something exits abnormally, but we'll see if that poses a problem in practice. </p>
</blockquote>
<p>This could be solved by throwing some extra tokens into the pipe, then discarding it and replacing it with a new one whenever a child doesn't exit properly. Old jobs will run on whatever tokens are left in the pipe and among each other and new ones will have a clean pool. It'll lead to temporary oversubscription but prevent permanent underutilization.</p>



<a name="235576008"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235576008" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235576008">(Apr 21 2021 at 20:29)</a>:</h4>
<p>Yeah, ultimately, it didn't really seem to hit that (at least how I'd expect it to if it happened), not sure whether that's due to a) my actually getting the implementation wrong somehow or b) few crates getting killed in a way that caused token loss. The load on the server definitely dropped, but this had no observable effect on actual throughput in terms of crates per minute or whatever. We can try again once the jobserver patch you posted lands and propagates enough, but that will also need a bump on the kernel for that machine (shouldn't be a problem, in theory, just needs some rebooting and such). We have discussion of those bumps scheduled for next week.</p>



<a name="235721244"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235721244" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235721244">(Apr 22 2021 at 17:58)</a>:</h4>
<p>FWIW currently experimenting with just reducing the thread count on the machines (i.e. how many parallel cargos etc we run), and at least initial results suggest that there's no difference between 10 and 5, and potentially no difference between 5 and 3 in terms of throughput, though average CPU usage definitely goes down.</p>
<p>That seems to suggest that throughput is blocked on some other resource, one we've not yet determined, but it seems hard to say definitively.</p>



<a name="235721367"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235721367" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235721367">(Apr 22 2021 at 17:58)</a>:</h4>
<p>(We do see larger caches, by virtue of splitting the disk amongst less workers, so maybe that is the tradeoff we're addressing)</p>



<a name="235721463"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235721463" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235721463">(Apr 22 2021 at 17:59)</a>:</h4>
<p><span class="user-mention" data-user-id="121055">@Pietro Albini</span> I'm curious if you know whether we had throughput metrics dictating the thread=10 count on the GCP workers, or if it was intended simply to ensure we were constantly at 100% CPU?</p>



<a name="235726053"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235726053" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235726053">(Apr 22 2021 at 18:29)</a>:</h4>
<p>that was a <em>long</em> time ago</p>



<a name="235726189"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235726189" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235726189">(Apr 22 2021 at 18:30)</a>:</h4>
<p>I think I guesstimated two years ago what would 100% the CPU and then did proportions since then</p>



<a name="235727582"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235727582" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235727582">(Apr 22 2021 at 18:39)</a>:</h4>
<p>ok</p>



<a name="235728263"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235728263" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235728263">(Apr 22 2021 at 18:43)</a>:</h4>
<p><a href="#narrow/stream/242791-t-infra/topic/playground.20issues.3F/near/234586123">https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/playground.20issues.3F/near/234586123</a><br>
nagisa suggested there's a prometheus exporter for pressure stall information. if you haven't looked at memory/IO pressure yet.</p>



<a name="235730105"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235730105" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235730105">(Apr 22 2021 at 18:57)</a>:</h4>
<p>Both seem ~zero, based on manual inspection on all of the runners.</p>



<a name="235730986"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235730986" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235730986">(Apr 22 2021 at 19:03)</a>:</h4>
<p>odd, why wouldn't it scale with cpu utilization then?</p>



<a name="235731154"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235731154" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235731154">(Apr 22 2021 at 19:04)</a>:</h4>
<p>how much is spent in the kernel?</p>



<a name="235731156"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235731156" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235731156">(Apr 22 2021 at 19:04)</a>:</h4>
<p>My guess is we're burning CPU, e.g., we have memory left (so no memory pressure) but are still hitting a bunch of overhead from context switching or whatever. But I don't know how to prove that.</p>



<a name="235731199"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235731199" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235731199">(Apr 22 2021 at 19:04)</a>:</h4>
<p>~10% 'sys' cpu usage</p>



<a name="235731817"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235731817" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235731817">(Apr 22 2021 at 19:09)</a>:</h4>
<div class="codehilite"><pre><span></span><code># on 3-worker node
$ tail /proc/pressure/*
==&gt; /proc/pressure/cpu &lt;==
some avg10=5.87 avg60=11.80 avg300=13.44 total=803265577317

==&gt; /proc/pressure/io &lt;==
some avg10=0.38 avg60=0.80 avg300=0.70 total=15699602295
full avg10=0.26 avg60=0.66 avg300=0.54 total=2398199866

==&gt; /proc/pressure/memory &lt;==
some avg10=0.00 avg60=0.00 avg300=0.00 total=2619255506
full avg10=0.00 avg60=0.00 avg300=0.00 total=444665793

# on a 10-worker node
$ tail /proc/pressure/*
==&gt; /proc/pressure/cpu &lt;==
some avg10=97.21 avg60=93.02 avg300=91.75 total=1463334511136

==&gt; /proc/pressure/io &lt;==
some avg10=0.90 avg60=1.34 avg300=1.31 total=27106684130
full avg10=0.00 avg60=0.00 avg300=0.00 total=3621447640

==&gt; /proc/pressure/memory &lt;==
some avg10=0.17 avg60=0.12 avg300=0.15 total=4874537937
full avg10=0.00 avg60=0.00 avg300=0.00 total=805547034
</code></pre></div>



<a name="235732323"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235732323" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235732323">(Apr 22 2021 at 19:13)</a>:</h4>
<p>well, that's still oversubscribing the CPU. CPU pressure should be 0 if there are exactly as many runnable threads as cores. you said you added a jobserver, so something isn't under jobserver control.</p>



<a name="235732367"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235732367" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235732367">(Apr 22 2021 at 19:13)</a>:</h4>
<p>this is without the jobserver</p>



<a name="235732389"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235732389" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235732389">(Apr 22 2021 at 19:13)</a>:</h4>
<p>I don't think I collected these stats with the jobserver</p>



<a name="235732614"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235732614" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235732614">(Apr 22 2021 at 19:15)</a>:</h4>
<p>we could of course try it again, I suppose</p>



<a name="235733070"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235733070" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235733070">(Apr 22 2021 at 19:18)</a>:</h4>
<p>slight oversubscription to keep the CPUs fed + jobserver to keep contention down seems like it would provide good throughput in theory</p>



<a name="235733130"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235733130" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235733130">(Apr 22 2021 at 19:19)</a>:</h4>
<p>ah, does that also run tests?</p>



<a name="235733141"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235733141" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235733141">(Apr 22 2021 at 19:19)</a>:</h4>
<p>the current jobs are just cargo check</p>



<a name="235733156"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/235733156" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#235733156">(Apr 22 2021 at 19:19)</a>:</h4>
<p>I don't think libtest uses jobserver today, so that's a valid point</p>



<a name="238916702"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/238916702" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#238916702">(May 15 2021 at 18:30)</a>:</h4>
<p>An interesting observation, though I'm not sure how reliable, is that it seems like the crater-azure instances <em>sped up</em> after gcp instances went offline. Not really sure what to make of this; I guess one possibility is we actually have a bottleneck in the rust-bots machine (or network to it, whatever).</p>
<p><a href="/user_uploads/4715/yIjl6ZQZJwUsCaWPOUciUJ58/image.png">image.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/yIjl6ZQZJwUsCaWPOUciUJ58/image.png" title="image.png"><img src="/user_uploads/4715/yIjl6ZQZJwUsCaWPOUciUJ58/image.png"></a></div>



<a name="238916773"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crater%20performance/near/238916773" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crater.20performance.html#238916773">(May 15 2021 at 18:31)</a>:</h4>
<p>it's possible we switched to a different job, too, I guess... not sure how reasonable that is</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>